feat(cli): emit telemetry for the validate action - #1864
Open
iankhou wants to merge 6 commits into
Open
Conversation
Adds a VALIDATE telemetry event covering the validation phase of cdk validate (offline report collection and online CloudFormation validation, excluding synthesis), with counters for offline violations per severity, offlineWouldFailDeploy (offline validation found a report that would have failed cdk deploy), and onlineViolations.
Contributor
Dependency Review✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.Scanned FilesNone |
iankhou
changed the title
- Remove the pre-synthesis before the VALIDATE span: it made the user-visible 'Synthesis time' message report the cache hit (~0s) instead of the real duration, and it moved app-crash-during-synth outside the span. Synthesis now runs inside the VALIDATE span; the SYNTH event (instrumented in CloudExecutable) stays accurate. - Extract wouldFailDeploy() from throwIfValidationFailures and use it for the offlineWouldFailDeploy counter, instead of re-deriving only the default failAt branch. - Only set ValidateResult.onlineReports when online validation ran, so undefined distinguishes 'skipped' from 'ran clean' as documented. - Deduplicate sum() into util/arrays instead of a third private copy.
Uses the validate-app fixture (SecurityPlugin + construct annotation) to assert the VALIDATE event and all four counter kinds in the telemetry file, and asserts the batch POST succeeds against the real endpoint, which doubles as an end-to-end check that the telemetry backend accepts the VALIDATE event type.
…nter key Newer aws-cdk-lib normalizes the fixture plugin's non-standard 'cost-optimization' severity to 'custom', so assert the stable counter keys plus the total offline violation count instead of that key.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
The
cdk validateaction (unstable) runs offline and online validation but currently emits no telemetry. This adds aVALIDATEtelemetry event so we can measure, at minimum, that the validation engine ran (start time, duration, success/failure), and whether offline validation caught errors that would have failed a deployment.Not included
cdk validate --watchdoes not emit VALIDATE events.We have, separately, enabled our Telemetry backend to accept VALIDATE events.
Checklist
count-validation-results.test.ts,cli-io-host.test.ts,validate.test.ts+ io-snapshots)By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license